Enhanced PeerHunter: Detecting Peer-to-peer Botnets through Network-Flow Level Community Behavior Analysis

Peer-to-peer (P2P) botnets have become one of the major threats in network security for serving as the fundamental infrastructure that responsible for various cyber-crimes. More challenges are involved in the problem of detecting P2P botnets, despite a few existing works claimed to detect traditional botnets effectively. In this paper, we present Enhanced PeerHunter, a network-flow level botnet community behavior analysis based method, which is capable of detecting botnets that communicate via P2P overlay networks. Our method starts from a P2P network flow detection component. Then, it uses the natural botnet behavior “mutual contacts” as the main feature to cluster bots into communities. Finally, it uses network-flow level botnet community behavior analysis to detect potential botnet communities and further identify bot candidates. In the experimental evaluation, we propose two evasion attacks, in which we assume the adversaries (e.g., the botmasters) know our techniques in advance, and they might attempt to evade our system via making the P2P bots mimic the behavior of legitimate P2P applications. The extensive experiments’ results show that Enhanced PeerHunter can achieve high detection rate with few false positives, and high robustness against the proposed mimicking legitimate P2P application attacks.